When lossy video compression is used, artifacts can be visible in decoded video. Use of high quantization values (i.e., coarse quantizations) reduces the bit usage but also makes quantization noise (i.e., coding noise) more visible. For a given quantization level, quantization noise is commonly more noticeable in smooth macroblocks and less noticeable in macroblocks with many pixel-to-pixel variations.
Referring to FIG. 1, a block diagram of a portion of a conventional video encoder 10 is shown. The conventional video encoder 10 (i.e., a compressor) can utilize conventional macroblock adaptive quantization (AQP) to adjust the quantization levels. A controller 12 of the encoder 10 makes measurements to determine a degree to which a macroblock will “show” quantization noise after being decoded. Typically, the measurements look at aspects of luminance sample variations and use quantization values that increase with the amount of the variations. A macroblock encoder 14 of the video encoder 10 then uses the quantization values to encode the macroblocks to generate a compressed bitstream. Another example of a conventional video encoder is a software encoder called Test Model 5. The Test Model 5 uses a block-variance based measure.
A linear quantization performed by a conventional encoder may be related to a quantization parameter. A linear quantization value (i.e., LINEAR_QUANT) is a number proportional to a quantization step size. In an H.264/AVC standard, a quantization parameter (i.e., QP) is an integer used to describe the quantization of a macroblock. The quantization parameter QP is related to the linear quantization by equation 1 as follows:LINEAR_QUANT=2^(QP/6)  Eq. (1)
Conventional adaptive quantization parameter (AQP) methods are based on an implicit model. The implicit models set the quantization parameter QP based on measured statistics to more or less equalize the visibility of quantization noise across all macroblocks. Because the models are not always accurate, the variations in minimum linear equalization to maximum linear quantization has to be limited. For example, a formula used in the Test Model 5 for AQP limits the ratio of maximum linear equalization to minimum linear quantization to 4:1 and, for normal video, the spread is typically much small (i.e., about 2:1). Even so, sequences exist where even a smaller spread in linear quantization results in better quality. Furthermore, sequences exist where a moderate spread in the linear quantization is not sufficient to provide adequate quality to very smooth macroblocks. For example, using just the measure and formulae from Test Model 5 produces a spread of 2:1, whereas a spread of 5:1 truly makes the smooth macroblocks look good.